Le rôle fondamental de l’observation directe dans la décision de confier une responsabilité professionnelle

Background: Entrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE). Methods: Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity. Results: Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases (p = <0.001). Conclusions: Ratings based on direct observation made unique contributions to entrustment decisions.Contexte : La décision de confier une activité peut être rétrospective (basée sur les expériences antérieures avec un apprenant) ou en temps réel (basée sur l’observation directe). Nous avons étudié les évaluations de niveaux de confiance fondées sur des interactions antérieures des candidats par les évaluateurs et celles fondées sur l’observation directe systématique, dans le cadre d’un examen clinique objectif structuré (ECOS). Méthodes : Seize évaluateurs du corps professoral ont fourni 287 évaluations rétrospectives et en temps réel du niveau de confiance faites lors des stations d’ECOS en 2019 et 2020 concernant 16 stagiaires en cardiologie. La fiabilité et la validité de ces évaluations ont été analysées en comparant les corrélations entre les stations comme mesure de la fiabilité, les différences entre les années d’études postdoctorales comme indice de la validité de construit, la corrélation avec l’examen normalisé en cours de formation (ITE) comme mesure de la validité de critère, et le reclassement des évaluations de la confiance comme mesure de la validité corrélative. Résultats : Les évaluations rétrospectives et en temps réel étaient toutes les deux très fiables (toutes les corrélations intra-classes >0,86). Les deux augmentaient avec le niveau de formation postdoctorale. Les évaluations de la confiance en temps réel étaient significativement corrélées aux scores de l’examen normalisé en cours de formation; les évaluations rétrospectives ne l’étaient pas. Les évaluations en temps réel expliquaient 37 % (2019) et 46 % (2020) de la variance des notes d’examen, contre 21 % (2019) et 7 % (2020) pour les évaluations rétrospectives. L’observation directe a permis de reclasser 44 % des évaluations rétrospectives de la confiance (p=<0,001 dans les deux cas). Conclusion : Les évaluations basées sur l’observation directe contribuent de façon importante à la décision de confier une activité

Enriched biodiversity data as a resource and service

Background: Recent years have seen a surge in projects that produce large volumes of structured, machine-readable biodiversity data. To make these data amenable to processing by generic, open source “data enrichment” workflows, they are increasingly being represented in a variety of standards-compliant interchange formats. Here, we report on an initiative in which software developers and taxonomists came together to address the challenges and highlight the opportunities in the enrichment of such biodiversity data by engaging in intensive, collaborative software development: The Biodiversity Data Enrichment Hackathon. Results: The hackathon brought together 37 participants (including developers and taxonomists, i.e. scientific professionals that gather, identify, name and classify species) from 10 countries: Belgium, Bulgaria, Canada, Finland, Germany, Italy, the Netherlands, New Zealand, the UK, and the US. The participants brought expertise in processing structured data, text mining, development of ontologies, digital identification keys, geographic information systems, niche modeling, natural language processing, provenance annotation, semantic integration, taxonomic name resolution, web service interfaces, workflow tools and visualisation. Most use cases and exemplar data were provided by taxonomists. One goal of the meeting was to facilitate re-use and enhancement of biodiversity knowledge by a broad range of stakeholders, such as taxonomists, systematists, ecologists, niche modelers, informaticians and ontologists. The suggested use cases resulted in nine breakout groups addressing three main themes: i) mobilising heritage biodiversity knowledge; ii) formalising and linking concepts; and iii) addressing interoperability between service platforms. Another goal was to further foster a community of experts in biodiversity informatics and to build human links between research projects and institutions, in response to recent calls to further such integration in this research domain. Conclusions: Beyond deriving prototype solutions for each use case, areas of inadequacy were discussed and are being pursued further. It was striking how many possible applications for biodiversity data there were and how quickly solutions could be put together when the normal constraints to collaboration were broken down for a week. Conversely, mobilising biodiversity knowledge from their silos in heritage literature and natural history collections will continue to require formalisation of the concepts (and the links between them) that define the research domain, as well as increased interoperability between the software platforms that operate on these concepts

Optimizing self-regulation of performance : is mental effort a cue?

Accurate self-regulation of performance is important for trainees. Trainees rely on cues to make monitoring judgments to self-regulate their performance. Ideally, cues and monitoring judgements accurately reflect performance, as measured by cue diagnosticity (the ability of a cue to predict performance) and monitoring accuracy (the ability of a monitoring judgement to predict performance). However, this process is far from perfect, emphasizing the need for more accurate cues and monitoring judgements. Perhaps the mental effort of a task could be a cue used to inform certainty judgements. The purpose of this study is to measure cue utilization and cue diagnosticity of mental effort and monitoring accuracy of certainty for self-regulation of performance. Focused on the task of ECG interpretation, 22 PGY 1-3 Internal Medicine residents at McMaster University provided a diagnosis for 10 ECGs, rating their level of certainty (0–100%) and mental effort (Paas scale, 1–9). 220 ECGs completed by 22 participants were analyzed using path analysis. There was a negative moderate path coefficient between certainty and mental effort (β = − 0.370, p < 0.001), reflecting cue utilization. Regarding cue diagnosticity of mental effort, this was reflected in a small negative path coefficient between mental effort and diagnostic accuracy (β = − 0.170, p = 0.013). Regarding monitoring accuracy, a moderate path coefficient was observed between certainty and diagnostic accuracy (β = 0.343, p < 0.001). Our results support mental effort as a cue and certainty as a monitoring judgement for self-regulated performance. Yet, reported correlations are not very high. Future research is needed to identify additional cues

The critical role of direct observation in entrustment decisions

Background: Entrustment decisions may be retrospective (based on past experiences with a trainee) or real-time (based on direct observation). We investigated judgments of entrustment based on assessor prior knowledge of candidates and based on systematic direct observation, conducted in an objective structured clinical exam (OSCE).Methods: Sixteen faculty examiners provided 287 retrospective and real-time entrustment ratings of 16 cardiology trainees during OSCE stations in 2019 and 2020. Reliability and validity of these ratings were assessed by comparing correlations across stations as a measure of reliability, differences across postgraduate years as an index of construct validity, correlation to standardized in-training exam (ITE) as a measure of criterion validity, and reclassification of entrustment as a measure of consequential validity.Results: Both retrospective and real-time assessments were highly reliable (all intra-class correlations >0.86). Both increased with year of postgraduate training. Real-time entrustment ratings were significantly correlated with standardized ITE scores; retrospective ratings were not. Real-time ratings explained 37% (2019) and 46% (2020) of variance in examination scores vs. 21% (2019) and 7% (2020) for retrospective ratings. Direct observation resulted in a different level of entrustment compared with retrospective ratings in 44% of cases (p = 0,86). Les deux augmentaient avec le niveau de formation postdoctorale. Les évaluations de la confiance en temps réel étaient significativement corrélées aux scores de l’examen normalisé en cours de formation; les évaluations rétrospectives ne l’étaient pas. Les évaluations en temps réel expliquaient 37 % (2019) et 46 % (2020) de la variance des notes d’examen, contre 21 % (2019) et 7 % (2020) pour les évaluations rétrospectives. L’observation directe a permis de reclasser 44 % des évaluations rétrospectives de la confiance (p=<0,001 dans les deux cas).Conclusion : Les évaluations basées sur l’observation directe contribuent de façon importante à la décision de confier une activité

GBIF Integration of Open Data

The Global Biodiversity Information Facility (GBIF) runs a global data infrastructure that integrates data from more than 1700 institutions. Combining data at this scale has been achieved by deploying open Application Programming Interfaces (API) that adhere to the open data standards provided by Biodiversity Information Standards (TDWG). In this presentation, we will provide an overview of the GBIF infrastructure and APIs and provide insight into lessons learned while operating and evolving the systems, such as long-term API stability, ease of use, and efficiency. This will include the following topics:The registry component provides RESTful APIs for managing the organizations, repositories and datasets that comprise the network and control access permissions. Stability and ease of use have been critical to this being embedded in many systems.Changes within the registry trigger data crawling processes, which connect to external systems through their APIs and deposit datasets into GBIF's central data warehouse. One challenge here relates to the consistency of data across a distributed network.Once a dataset is crawled, the data processing infrastructure organizes and enriches data using reference catalogues accessed through open APIs, such as the vocabulary server and the taxonomic backbone. Being able to process data quickly as source data and reference catalogues change is a challenge for this component.The data access APIs provide search and download services. Asynchronous APIs are required for some of these aspects, and long-term stability is a requirement for widespread adoption. Here we will talk about policies for schema evolution to avoid incompatible changes, which would cause failures in client systems.The APIs that drive the user interface have specific needs such as efficient use of the network bandwidth. We will present how we approached this, and how we are currently adopting GraphQL as the next generation of these APIs. There are several APIs that we believe are of use for the data publishing community. These include APIs that will help in data quality aspects, and new data of interest thanks to the data clustering algorithms GBIF deploys